American Journal of Business Education - March/April 2009 


Volume 2, Number 2 


Retention Assessment Of Core Operations 
Management Topics For Business 
Administration Students 

Nicole B. Koppel, Montclair State University, USA 
Kimberly Killmer Hollister, Montclair State University, USA 


ABSTRACT 

To meet the new AACSB International standards regarding retention assessment and adequately 
determine “if and what students are learning, ” this research presents a framework within which 
expected learning outcomes and specific learning are assessed. This paper presents the 
framework and describes how the process can be implemented with an application to retention 
assessment for the core operations management (OM) topics. Assessing core OM topics partially 
fulfills the AACSB International mandate of topical coverage and outcome assessment in the area 
of analytical decision making, which includes business statistics, financial analysis, and 
operations management. The study is of value to those who desire to better understand and 
implement assurance of learning or assessment in their programs. 
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INTRODUCTION 


yy 'n recent years there has been a growing public dissatisfaction with the quality of college graduates 
(Martell and Calderon, 2005). In response, accrediting bodies, such as the Association to Advance 

5_ S Collegiate Schools of Business (AACSB) International, have adopted and revised their accreditation 

standards with a main focus on “assurance or learning (AoL)” (AASCB International, 2005). In order to achieve or 
re-achieve AACSB International accreditation, schools must have an assessment plan that addresses standards set 
forth as AoL Standards (White, 2007). Accredited schools are required to periodically assess student progress on 
key learning goals and to use the results to strengthen the curriculum as part of their continuing improvement efforts. 


Traditionally, university programs have been evaluated based on course curriculum — the specific material 
that instructors purport to teach, faculty qualifications, and admission standards. Business programs are now 
required to demonstrate students’ accomplishment levels in terms of established learning goals. Programs are 
required to provide evidence, not merely intent. The new assessment standard has led faculty and administrators to 
search for ways to collect evidence that university students are, in fact, learning and remembering what they are 
taught. 


This focus on retention assessment differs from the traditional grading process. Traditional grading has 
focused on assessing student classroom activities, effort, and performance. Retention assessment focuses on 
producing tangible evidence of knowledge acquisition or skill development (Mackenzie, 2004). With this increased 
emphasis on retention assessment and assurance of learning standards, more attention is being placed on the 
development and content of the curriculum through requiring specific knowledge and skills. The AACSB standards 
mandate topical coverage and outcome assessments in areas such as (a) communication, (b) ethics, (c) analytical 
decision making, (d) the use of information technology, (e) value creation chains, (f) globalization, (g) 
organizational leadership, and (h) appreciation for diversity and multicultural issues (Miles, Hazeldine, and Munilla 
2004). 
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Three key issues can be identified in the assessment process: (1) what learning needs to be assessed and to 
what degree; (2) how to measure and demonstrate achievement of learning goals (Zhu and McFarland, 2005) and (3) 
how to use those findings to strengthen the curriculum. According to Martell and Calderon (2005), in order to 
develop specific assessment measures, the AACSB International requires learning outcomes to be defined for each 
program which incorporate student mastery of knowledge and skills in these areas. These learning outcomes are 
used to establish specific learning goals which are the concrete observable behaviors and actions that can be 
measured. 



Figure 1: A Framework for Assessing Knowledge 


This research paper focuses on a framework within which an undergraduate program at an AACSB 
International-accredited business school incorporates retention assessment in the undergraduate business 
administration program. The paper discusses the motivation for the assessment program, how the assessment is 
implemented, and at what point in the undergraduate program assessment is incorporated. The framework is 
illustrated with a case study in the retention assessment of core operations management topics. These include those 
topics on OM that all business students must understand and retain, regardless of their selected concentration (i.e. 
accounting, marketing, etc.). 

FRAMEWORK FOR ASSESSMENT 

The development of a program to assess learning goals is complex. Business programs need a stepwise 
process to follow to direct their efforts in the assessment arena. In Figure 1 we describe a framework (Hollister and 
Koppel, 2005) for the assessment of student knowledge. The remainder of this section describes each stage in the 
process. 


The first step in the process is the identification of key Stakeholders, both internal and external. Every 
business school has a unique niche in the market; Stakeholder inputs drive the development of the Strategic 
Charter/Mission for each school. Based upon input from Stakeholders and the Strategic Charter, programs need to 
develop their Expected Learning Outcomes. Expected Learning Outcomes should be viewed as the broad areas in 
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The next step in the process is the development of specific learning goals for students. We break learning 
goals down into two categories: Degree Specific and Discipline Specific. We view the Degree Specific Learning 
Goals as learning goals that will be covered in either the Required General Education Courses or the Business Core 
Courses; courses that every business student will take regardless of concentration. Discipline Specific Learning 
Goals are related to the particular concentration(s) that students select within the Business Administration Degree. 

Based upon Learning Goals defined for both the Degree and Concentration, Course Specific Learning 
Goals can be established. Course Specific Goals are translated into specific course outlines and delivered using 
various pedagogical techniques. Student learning occurs at the course level and learning outcomes are measured 
against course, discipline, and program learning goals. Evaluation of student learning should be performed; at the 
program and discipline level to refine curriculum and at the individual level to provide feedback that can be used to 
reinforce the learning of specific knowledge and skills and to assist the students in managing their learning progress. 

The framework is complex because of the relationships among the three levels (Degree, Discipline, and 
Course); a change in the goals or techniques at any level initiates change at the other levels. For instance, the 
change in topics in a core course will impact both concentration and Degree Specific Learning Goals; similarly, a 
change in a Degree Specific Learning Goal will require changes in content of specific courses. 

One aspect of assurance of learning is the realization that students may pass courses but fail to demonstrate 
knowledge in some subject areas. Under the new assessment guidelines, these students are required to repeat the 
learning process until they reach a satisfactory level of performance. Overall, the assessment process is a closed- 
loop approach for continuous improvement to the program and assurance of learning. The emphasis is on learning. 

As the AACSB International standard on AoL is relatively new, the body of literature discussing overall 
program assessment and discipline specific concentration assessment is rather new. Most of the literature 
documents methodologies that institutions are piloting to move towards meeting this new goal. Pringle and Michel 
(2007) and Martell (2007) provide overviews of current assessment practices. Drinka et al (2005) discus an online 
assessment system they develop to assist in data collection and analysis. Bycio and Allen (2004) delve into 
assessment through a student survey; AACSB International has stated that student outcome surveys are not 
sufficient to demonstrate assurance of learning. Black and Duhon (2003) explore the controversial issue of using 
standardized assessment tests at the undergraduate level. 

As discussed above, AACSB International requires assessment in at least eight topical areas. Some of the 
literature focuses on assessment within these areas. As with the literature on overall program assessment, this 
literature is still in its infancy. Ali and Ho (2007) investigate the impact of various forms of course assessment on 
student performance. Fraser et al. (2005) discuss assessment in the writing aspect of communication. Hindi and 
Miller (2000) report results of a survey of assessment techniques in the area of accounting, Boldt (2001) presents a 
structural approach towards assessing accounting knowledge, Daigle et al. (2007) assesses accounting information 
system skills through case studies and Reffell and Whitworth (2002) examine the information technology skills of 
students. 

CASE APPLICATION - OPERATIONS MANAGEMENT 

To illustrate how the proposed framework can be used to develop an assessment program in a business 
school we present an application of the framework to the retention assessment of students’ knowledge of the core 
operations management skills for all business students. Assessing core OM topics partially fulfills the AACSB 
International mandate of topical coverage and outcome assessment in the area of analytical decision making, which 
includes business statistics, financial analysis, and operations management. 
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In order to achieve the Montclair State University’s School of Business goal of “graduating students who 
are immediately effective in cutting-edge business organizations” candidates for the degree of Bachelor of Science 
in Business Administration must have a firm understanding of the essential concepts of operations management and 
be good problem solvers so that they are better prepared to assist in the decision-making process. 

Another element in the Strategic Charter is the identification of “specific stakeholders as significant 
partners to our success as a school of business.” In developing expected learning outcomes for our graduating 
students, we consulted with a number of our outside stakeholders as well as internal stakeholders, SBUS faculty, to 
develop our objectives. To this end, we consulted with textbook authors, employers, the business and professional 
community, and the greater academic community to identify broad objectives that each graduating student should 
retain. 


We identify four broad expected learning outcomes relating to operations management and analytical 
decision making: strategy, monitoring and controlling processes, modeling, and solving business problems. 


Learning Outcome 1: 
Learning Outcome 2: 
Learning Outcome 3: 
Learning Outcome 4: 


Students will be able to understand the role Operations Management plays in making 
strategic decisions. 

Students will be able to use appropriate methods to monitor and control production and 
service processes. 

Students will be able to develop appropriate mathematical models to represent real-world 
situations. 

Students will be able to evaluate and solve business problems. 


SPECIFIC LEARNING GOALS 


The next step in the process is the identification of specific learning goals to assess; we are focusing on 
degree specific learning goals not learning goals associated with a specific concentration. We can also think of these 
as the specific competencies that will be evaluated. Specific learning goals are developed for each of the four 
learning outcomes identified in the previous section. For clarity, we present each of the four broad learning 
outcomes followed by a list with specific learning goals for that outcome. 

Learning Outcome 1 

Students Will Be Able To Understand The Role Operations Management Plays In Making Strategic 
Decisions. 


o Students must be able to understand the role of a supply chain and methods to improve it. 

o Students must be able to understand the importance of reducing costs throughout the supply chain 

o Students must be able to understand the strategic role in the design of goods and services 

o Students must be able to understand the strategic role in the selection of a location and layout of an 

operation 

o Students must be able to understand Project Management techniques 

Learning Outcome 2 

Students Will Be Able To Use Appropriate Methods To Monitor And Control Production And Service 
Processes. 


o Students must be able to understand how to monitor quality and ensure Total Quality Management (TQM), 

o Students must be able to understand how to select, implement, and monitor appropriate forecasting 

techniques. 

o Students must be able to understand how to monitor and control inventory levels. 
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o Students must be able to understand how to monitor and control job scheduling, 

o Students must be able to understand how to monitor and control waiting-line (queuing). 

Learning Outcome 3 

Students will be able to develop appropriate mathematical models to represent real-world situations. 

o Students must be able to develop appropriate mathematical programming models 

o Students must be able to develop decision making models 

Learning Outcome 4 

Students will be able to evaluate and solve business problems. 

o Students must be able to determine which information is relevant to analyze business problems 

o Students must be able to select an appropriate methodology for analysis and develop a model, if necessary 

o Students must be able to implement the methodology and evaluate alternatives 

o Students must be able to select best alternative, implement it and monitor results 

MEASURING LEARNING GOALS: RETENTION ASSESSMENT TEST 

Once learning outcomes and goals are developed, the next step in the process is the development of an 
assessment tool to measure student knowledge. Retention assessment tests are strategically placed in the business 
administration curriculum to ensure that students have the requisite knowledge to take advanced courses and be 
immediately effective in the workplace. Prior to enrolling in the upper-level business courses, students need to 
demonstrate that they possess those skills that were taught in the pre-requisite courses and are assumed to be known. 

The retention assessment test evaluates specific learning outcomes and goals that should be retained by 
students as they prepare to obtain their Business Administration degree. The exam is less comprehensive than what 
one might expect to see on a final exam for a course. Only major concepts are included and the expectation for 
passing the assessment test is commensurate with the skill level of an academically average student. 

To ensure that students graduate with competency in their core operations management concepts and 
problem solving skills, students will be required to take the OM Retention Assessment Instrument prior to 
registering for their capstone business course. It is strongly recommended that students take this course during their 
last semester before graduation. 

The OM Retention Assessment Instrument is a concepts-based exam comprised of both multiple choice and 
open-ended questions. Students can earn partial credit for the latter. A detailed rubric is developed for each of these 
questions with points awarded for the completeness of the answer and/or analysis. 

CLOSING THE LOOP - PROVIDING FEEDBACK 

The goal of assessment is to improve student learning. The assessment process requires us to broaden our 
thinking from being course-centered to focusing on the Business Administration program as a whole. Although we 
have worked through the process from top to bottom, we are not yet done. We must now begin the more difficult 
task of examining the results of the retention assessment test and closing the loop by making the required 
adjustments to our curriculum. In our framework. Figure 1, this is depicted as feedback from Measuring Learning 
Goals back up to all levels of the process from examining our strategic charter down to looking at our course 
delivery system. The goal is continual improvement. 

The AACSB International assessment standards, requiring evidence that business students are developing 
the skills and knowledge base that the curriculum purports to teach, coupled with the need to evaluate and address 
the knowledge retention issue, have motivated the SBUS at Montclair State University to incorporate a retention 
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assessment program in the undergraduate business curriculum. 

Both AACSB International and the SBUS at Montclair State University recognize the ability to understand 
core Operations Management concepts and develop problem solving skills in a business environment as a necessity 
for a graduate in Business Administration. The framework for OM assessment presented in this paper describes one 
approach to developing and implementing a program to meeting the goal of assurance of learning. 

It is expected that this assessment program will enable the SBUS at Montclair State University to truly 
achieve its goal of “graduating students who are immediately effective in cutting-edge business organizations.” 
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